Remote Sensing Data Analytics with the Udocker Container Tool using Multi-GPU Deep Learning Systems

Multi-GPU systems are in continuous development to deal with the challenges of intensive computational big data problems. On the one hand, parallel architectures provide a tremendous computation capacity and outstanding scalability. On the other hand, the production path in multi-user environments faces several roadblocks since they do not grant root privileges to the users. Containers provide flexible strategies for packing, deploying and running isolated application processes within multi-user systems and enable scientific reproducibility. This paper describes the usage and advantages that the uDocker container tool offers for the development of deep learning models in the described context. The experimental results show that uDocker is more transparent to deploy for less tech-savvy researchers and allows the application to achieve processing time with negligible overhead compared to an uncontainerized environment

A container-based workflow for distributed training of deep learning algorithms in HPC clusters

Deep learning has been postulated as a solution for numerous problems in different branches of science. Given the resource-intensive nature of these models, they often need to be executed on specialized hardware such graphical processing units (GPUs) in a distributed manner. In the academic field, researchers get access to this kind of resources through High Performance Computing (HPC) clusters. This kind of infrastructures make the training of these models difficult due to their multi-user nature and limited user permission. In addition, different HPC clusters may possess different peculiarities that can entangle the research cycle (e.g., libraries dependencies). In this paper we develop a workflow and methodology for the distributed training of deep learning models in HPC clusters which provides researchers with a series of novel advantages. It relies on udocker as containerization tool and on Horovod as library for the distribution of the models across multiple GPUs. udocker does not need any special permission, allowing researchers to run the entire workflow without relying on any administrator. Horovod ensures the efficient distribution of the training independently of the deep learning framework used. Additionally, due to containerization and specific features of the workflow, it provides researchers with a cluster-agnostic way of running their models. The experiments carried out show that the workflow offers good scalability in the distributed training of the models and that it easily adapts to different clusters

A Container-Based Workflow for Distributed Training of Deep Learning Algorithms in HPC Clusters

Deep learning has been postulated as a solution for numerous problems in different branches of science. Given the resource-intensive nature of these models, they often need to be executed on specialized hardware such graphical processing units (GPUs) in a distributed manner. In the academic field, researchers get access to this kind of resources through High Performance Computing (HPC) clusters. This kind of infrastructures make the training of these models difficult due to their multi-user nature and limited user permission. In addition, different HPC clusters may possess different peculiarities that can entangle the research cycle (e.g., libraries dependencies). In this paper we develop a workflow and methodology for the distributed training of deep learning models in HPC clusters which provides researchers with a series of novel advantages. It relies on udocker as containerization tool and on Horovod as library for the distribution of the models across multiple GPUs. udocker does not need any special permission, allowing researchers to run the entire workflow without relying on any administrator. Horovod ensures the efficient distribution of the training independently of the deep learning framework used. Additionally, due to containerization and specific features of the workflow, it provides researchers with a cluster-agnostic way of running their models. The experiments carried out show that the workflow offers good scalability in the distributed training of the models and that it easily adapts to different clusters.Comment: Under review for Cluster Computin

Ozone assessment as an EOSC-Synergy thematic service

Ozone assessment is an important task for Climate and Environment studies. The ozone assessment service (O3as) project is going to support scientists and everyone interested in determining ozone trends for different parts of the world. It is one of the thematic services of the EOSC-Synergy project. The service applies a unified approach to analyse results from a large number of different chemistry-climate models, helps to harmonise the calculation of ozone trends efficiently and consistently, and produce publication-quality figures in a coherent and user-friendly way. Among other tasks it will aid scientists to prepare the quadrennial Global Assessment of Ozone depletion. It will also allow access to the high-level data by citizens. The service relies on several containerized components distributed across the cloud (Kubernetes) and HPC resources and leverages large scale data facility (LSDF)

An online service for analysing ozone trends within EOSC-synergy

The European Open Science Cloud-Synergy (EOSC-Synergy) project delivers services that serve to expand the use of EOSC. One of these services, O3as, is being developed for scientists using chemistry-climate models to determine time series and eventually ozone trends for potential use in the quadrennial Global Assessment of Ozone Depletion, which will be published in 2022. A unified approach from a service like ours, which analyses results from a large number of different climate models, helps to harmonise the calculation of ozone trends efficiently and consistently. With O3as, publication-quality figures can be reproduced quickly and in a coherent way. This is done via a web application where users configure their queries to perform simple analyses. These queries are passed to the O3as service via an O3as REST API call. There, the O3as service processes the query and accesses the reduced dataset. To create a reduced dataset, regular tasks are executed on a high performance computer (HPC) to copy the primary data and perform data preparation (e.g. data reduction, standardisation and parameter unification). O3as uses EGI check-in (OIDC) to identify users and grant access to certain functionalities of the service, udocker (a tool to run Docker containers in multi-user space without root privileges) to perform data reduction in the HPC environment, and the Universitat Politècnica de València (UPV) Infrastructure Manager to provision service resources (Kubernetes)

Rootless containers with udocker

Trabajo presentado al ISC High Performance, celebrado en Frankfurt (Alemania) del 16 al 20 de junio de 2019.This work was developed with the support of the H2020 projects: INDIGO-DataCloud (RIA 653549), DEEP-Hybrid-DataCloud, (RIA 777435), EOSC-hub (RIA 777536). This work was developed with the support of the Portuguese Foundation for Science and Technology under the project: Infraestrutura Nacional de Computação Distribuída 01/SAICT/2016 - nº 0221

A survey of the European Open Science Cloud services for expanding the capacity and capabilities of multidisciplinary scientific applications

Open Science is a paradigm in which scientific data, procedures, tools and results are shared transparently and reused by society as a whole. The initiative known as the European Open Science Cloud (EOSC) is an effort in Europe to provide an open, trusted, virtual and federated computing environment to execute scientific applications, and to store, share and re-use research data across borders and scientific disciplines. Additionally, scientific services are becoming increasingly data-intensive, not only in terms of computationally intensive tasks but also in terms of storage resources. Computing paradigms such as High Performance Computing (HPC) and Cloud Computing are applied to e-science applications to meet these demands. However, adapting applications and services to these paradigms is not a trivial task, commonly requiring a deep knowledge of the underlying technologies, which often constitutes a barrier for its uptake by scientists in general. In this context, EOSC-SYNERGY, a collaborative project involving more than 20 institutions from eight European countries pooling their knowledge and experience to enhance EOSC\u27s capabilities and capacities, aims to bring EOSC closer to the scientific communities. This article provides a summary analysis of the adaptations made in the ten thematic services of EOSC-SYNERGY to embrace this paradigm. These services are grouped into four categories: Earth Observation, Environment, Biomedicine, and Astrophysics. The analysis will lead to the identification of commonalities, best practices and common requirements, regardless of the thematic area of the service. Experience gained from the thematic services could be transferred to new services for the adoption of the EOSC ecosystem framework

EOSC-SYNERGY EU Deliverable D2.3: Final report on EOSC integration

This report is delivered in the form of a "Handbook" on how to integrate national clouds, thematic resources, and data repositories conformant to common quality standards, and harmonised in terms of technological, policy, and legal aspects.EOSC-SYNERGY receives funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 857647.Peer reviewe

Search for top-down and bottom-up drivers of latitudinal trends in insect herbivory in oak trees in Europe

International audienceAim: The strength of species interactions is traditionally expected to increase toward the Equator. However, recent studies have reported opposite or inconsistent latitudinal trends in the bottom-up (plant quality) and top-down (natural enemies) forces driving herbivory. In addition, these forces have rarely been studied together thus limiting previous attempts to understand the effect of large-scale climatic gradients on herbivory. Location: Europe. Time period: 2018–2019. Major taxa studied: Quercus robur. Methods: We simultaneously tested for latitudinal variation in plant–herbivore–natural enemy interactions. We further investigated the underlying climatic factors associated with variation in herbivory, leaf chemistry and attack rates in Quercus robur across its complete latitudinal range in Europe. We quantified insect leaf damage and the incidence of specialist herbivores as well as leaf chemistry and bird attack rates on dummy caterpillars on 261 oak trees. Results: Climatic factors rather than latitude per se were the best predictors of the large-scale (geographical) variation in the incidence of gall-inducers and leaf-miners as well as in leaf nutritional content. However, leaf damage, plant chemical defences (leaf phenolics) and bird attack rates were not influenced by climatic factors or latitude. The incidence of leaf-miners increased with increasing concentrations of hydrolysable tannins, whereas the incidence of gall-inducers increased with increasing leaf soluble sugar concentration and decreased with increasing leaf C : N ratios and lignins. However, leaf traits and bird attack rates did not vary with leaf damage. Main conclusions: These findings help to refine our understanding of the bottom-up and top-down mechanisms driving geographical variation in plant–herbivore interactions, and indicate the need for further examination of the drivers of herbivory on trees